Blended Classic Joint Attention and Multimodal Deixis

Deixis and Conjunction in Multimodal Systems

In order to realize their full potential, multimodal interfaces need to support not just input from multiple modes, but single commands optimally distributed across the available input modes. A multimodal language processing architecture is needed to integrate semantic content from the different modes. Johnston 1998a proposes a modular approach to multimodal language processing in which spoken ...

Understanding multimodal deixis with gaze and gesture in conversational interfaces

Deixis in Multimodal Human Computer Interaction: An Interdisciplinary Approach

Focusing on deixis in human computer interaction this paper presents interdisciplinary work on the use of co-verbal gesture . Empirical investigations, theoretical modeling, and computational simulations with an anthropomorphic agent are based upon comparable settings and common representations. Findings pertain to the coordination of verbal and gestural constituents in deictic utterances. We d...

Incremental Generation of Multimodal Deixis Referring to Objects

This paper describes an approach for the generation of multimodal deixis to be uttered by an anthropomorphic agent in virtual reality. The proposed algorithm integrates pointing and definite description. Doing so, the context-dependent discriminatory power of the gesture determines the contentselection for the verbal constituent. The concept of a pointing cone is used to model the region single...

A Multimodal Interaction Framework for Blended Learning

Humans interact with each other by utilizing the five basic senses as input modalities, whereas sounds, gestures, facial expressions etc. are utilized as output modalities. Multimodal interaction is also used between humans and their surrounding environment, although enhanced with further senses such as equilibrioception and the sense of balance. Computer interfaces that are considered as a dif...